class: center, middle, inverse, title-slide .title[ # Systematic Literature Review ] .subtitle[ ## Part 1: Introduction ] .author[ ### Faisal Mustafa ] .date[ ### 2024-05-31 ] --- <h1> Outline of Part 1:</h1> <hr> <h2> This part of presentation covers:</h2> <ul> <li> Definition </li> <li> Research questions </li> <li> Steps of SLR research </li> <li> Writing an SLR paper </li> </ul> Slides are accessible at https://faisalmustafa.github.io/SLR/ <br> Important other resources are available <a href="https://github.com/faisalmustafa/SLR/tree/38bbafed72eb2f1fdc722d63518a17ca91625bc9/Recommended_readings"> HERE </a> <img class = "qr" src="images/qr-code.png" width = 300px alt="QR code of "> --- <h1> Definition </h1><hr class="vspace"> <h7> <em>A method of making sense of large bodies of information, and a means to contributing to the answers to questions about what works and what does not.</em></h7> <br><br> <table style="width:70%"> <tr> <th>Narrative LR</th> <th>Systematic LR</th> </tr> <tr> <td>Variety of styles</td> <td>Structured approach</td> </tr> <tr> <td>No defined method</td> <td>Rigorous method</td> </tr> <tr> <td>No specified analysis</td> <td>Synthesis to answer RQ / achieve research objectives</td> </tr> </table> <p class = "citation"> Jesson, J. K., Matheson, L., & Lacey, F. M. (2011). <em>Doing your systematic literature review: Traditional and systematic technique.</em> SAGE Publications Ltd,</p> --- <h1> Publishable SLR: <span class="subtitle"> To stand out of the review </span></h1><hr> -- ⛯ The objective of SLR: To answer the research question.<br><br> -- ⛯ Start SLR with research question: Be as specific as possible. <br>   Example 1 = https://doi.org/10.3389/fpsyg.2022.1075818 <br>   Example 2 = https://doi.org/10.1080/17501229.2024.2355296 <br><br> -- ⛯ Better format: <img src="images/SLR format.png" width = 75% alt="My SLR format">   Example 1 = https://doi.org/10.1016/j.ijer.2024.102380 <br>   Example 2 = https://doi.org/10.1080/17501229.2024.2355296 --- <h1>Steps in doing SLR</h1><hr> Step 1: Define the real objective (for publication) <br> Step 2: Decide research question <br> Step 3: Define inclusion and exclusion criteria <br> Step 4: Search the literature <br> Step 5: Screen the papers <br> Step 6: Retrieve full papers <br> Step 7: Conduct quality assessment <br> Step 8: Coding process <br> Step 9: Synthesis to achieve the research objective <br> Step 10: Writing up the SLR paper --- <h1> Real objective and RQ </h1><hr> <h2>My scenario:</h2> <span class="highlight"> Topic:</span> Innovation in digital transformation [scope = in Southeast Asian higher education] <br><br> <span class="highlight"> Gap:</span> But no clear definition of innovation <br><br> <span class="highlight">Objective:</span> Definition of innovation in DT <br><br> <span class="highlight"> Research questions:</span> What are the criteria of innovation in digital transformation in Southeast Asian higher education? <footer> Any questions so far? </footer> --- <h1> Inclusion criteria and literature search</h1><hr> <ol> <li> Databases: Scopus, Webs of Science, ProQuest, etc.</li> -- <li> Publication type: Peer reviewed such as journal articles and conference proceedings </li> -- <li> Period. Be clear of the reason </li> -- <li> Keywords: Teach yourselves Boolean AND, OR, AND NOT <br> Let's demonstrate at <a href="https://www.scopus.com"> https://www.scopus.com</a>, <a href="https://www.webofscience.com/wos/">https://www.webofscience.com/wos/</a>, and <a href="https://www.proquest.com">https://www.proquest.com </a> </li> <li> Including only open-access articles is not recommended </li> </ol> <img src="images/databases.png" width = 100% alt="Structuring research results" border="1px #555555"> <footer> Any questions so far? </footer> --- <h1> Lit. preparation, screening and retrieval</h1><hr> <footer> lit. = literature </footer> If excel is more preferred for final list preparation: <ul> <li> Sorting </li> <li> Conditional formatting </li> </ul> -- Screening process: <ul> <li> Read title, abstract, and keywords (but mostly abstract) </li> <li> Check type of publication because the database search might be incorrect </li> <li> Record reasons of exclusion </li> </ul> -- The full text of the final list can be obtained from: <ul> <li class = "highlight"> University library subscription </li> <li class = "highlight"> <em> Perpusnas</em> subscription </li> <li> Researchgate or other free repository </li> <li> Authors </li> </ul> <h8>All process and the number of publications resulted after each process must be recorded.</h8> --- <h1> Quality assessment:<span class="subtitle"> Quality appraisal of RAs</span></h1><hr> <ul class = "no-tab"> <li> To ensure that the coding process is smooth </li> <li> High quality articles provide all information we need for SLR <li> low quality papers mush be excluded from the review </ul> <h8> Framework for quality appraisal of research articles (RAs) (Nguyen, et al., 2024) </h8> <table class="blueTable"> <thead> <tr> <th style="width:87%">Criteria</th> <th style="width:13%; text-align: left;">Yes (1)   No (0)</th> </tr> </thead> <tbody> <tr> <td>1. Are the research questions (RQs) or objectives clearly and appropriately defined?</td><td></td></tr> <tr> <td>2. Is the research design appropriate for addressing the RQ(s)/objective(s) and clearly presented in the article?</td><td></td></tr> <tr> <td>3. Is the sampling strategy appropriately justified?</td><td></td></tr> <tr> <td>4. Does the article clearly describe the setting of data collection?</td><td></td></tr> <tr> <td>5. Is/Are the method(s) of data collection appropriate for addressing the RQ(s)/objective(s) and clearly presented in the article?</td><td></td></tr> <tr> <td>6. Is/Are the key concept(s) (e.g. PLC) clearly defined in the article?</td><td></td></tr> <tr> <td>7. Is/Are the method(s) of data analysis appropriate for addressing the RQ(s)/objective(s) and clearly presented?</td><td></td></tr> <tr> <td>8. Is/Are the research question(s) or objective(s) answered?</td><td></td></tr> <tr> <td>9. Are the discussion/conclusion(s)/implication(s) data appropriate?</td><td></td></tr> <tr> <td>10. Is there evidence of attention to ethical issues?</td><td></td></tr> <tr> <td class="bold">Total Score</td><td></td></tr> </tbody> </tr> </table> <p style="font-size: 17px"> <strong>Yes</strong> = Score 1; <strong>No</strong> = Score 0; <strong>High quality</strong> = 8-10; <strong>Medium quality</strong> = 5-7; <strong>Low quality</strong> = 0-4 </p> <p class="citation"> Nguyen, D., Boeren, E., Maitra, S., & Cabus, S. (2024). A review of the empirical research literature on PLCs for teachers in the Global South: Evidence, implications, and directions. <em>Professional Development in Education, 50</em>(1), 91–107. <a href="https://doi.org/10.1080/19415257.2023.2238728">https://doi.org/10.1080/19415257.2023.2238728</a> </p> --- <h1> Coding process </h1><hr> <h3> Phase A: Coding preparation </h3> <strong>Step 1:</strong> Determine framework <br> <strong>Step 2:</strong> Design a coding protocol <br> <span style="margin-left: 67px"> ⬥ Prioritize <em> Discussion → Conclusion → Results </em></span><br> <span style="margin-left: 67px"> ⬥ Code only one sentence or specific part of the sentence</span><br> <span style="margin-left: 67px"> ⬥ What sentence to code when the idea is expressed in one paragraph</span><br> <span style="margin-left: 67px"> ⬥ Etc.</span><br> <strong>Step 3:</strong> Recruit another coder (or more coders, depending on the size of the project) <br> <strong>Step 4:</strong> Socialize the framework and coding protocol to other coders --- <h1> Coding process </h1><hr> <h3> Phase B: Inter-coder reliability</h3> <img src="images/ICR.png" width = 73% alt="Inter-coder reliability"> <div class="topright"> <table class="blueTable" style ="width:25%"> <thead> <tr> <th style="width:10%; text-align: center;">Kappa statistic</th> <th style="width:15%; text-align: center;">Strength of Agreement</th> </tr> </thead> <tbody> <tr> <td style ="text-align:center"> <0.00 </td><td style ="text-align:center">Poor</td></tr> <tr> <td style ="text-align:center">0.00 - 0.20</td><td style ="text-align:center">Slight</td></tr> <tr> <td style ="text-align:center">0.21 - 0.40</td><td style ="text-align:center">Fair</td></tr> <tr> <td style ="text-align:center">0.41 - 0.60</td><td style ="text-align:center">Moderate</td></tr> <tr> <td style ="text-align:center">0.61 - 0.80</td><td style ="text-align:center">Substantial</td></tr> <tr> <td style ="text-align:center">0.81 - 1.00</td><td style ="text-align:center">Almost perfect</td></tr> </tbody> </table> </div> <p class="citation2"> Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. <em>Biometrics, 33</em>(1), 159-174. <a href="https://doi.org/10.2307/2529310"> https://doi.org/10.2307/2529310 </a></p> <p class="citation"> Rau, G., & Shih, Y. (2021). Evaluation of Cohen's kappa and other measures of inter-rater agreement for genre analysis and other nominal data. <em>Journal of English for Academic Purposes, 53</em>, Article 101026. <a href="https://doi.org/10.1016/j.jeap.2021.101026"> https://doi.org/10.1016/j.jeap.2021.101026 </a></p> --- <h1> Coding process </h1><hr> <h3> Part C: Coding </h3> <h7> Popular software for qualitative data coding in addition to Microsoft Excel:</h7> <table class="blueTable" style ="width:70%; float:left"> <thead style = "text-align: left"> <tr> <th style="width:5%">No</th> <th style="width:20%">Software</th> <th style="width:20%">Price (IDR)</th> <th style="width:30%;">License and duration</th> </tr> </thead> <tbody> <tr> <td> 1</td> <td> NVIVO </td><td>2,700,000</td><td> Student, 1 year</td></tr> <tr> <td> 2</td> <td> Atlas.ti </td><td>1,900,000</td><td> Student, 1 year</td></tr> <tr> <td> 3</td> <td> MAXQDA </td><td>1,700,000</td><td> Student, 2 year</td></tr> <td> </td> <td> </td><td>950,000</td><td> Student, 6 months</td></tr> <tr> <td> 4</td> <td> QualCoder </td><td>Open-source</td><td> -</td></tr> </table> <footer>Let's agree on the software and have a dedicated session on this part of coding process.</footer> --- <h1> Synthesis of the research results </h1><hr> <h8> The key point is to conclude the results and lead to the novelty or the real objective. </h8><br><br> <img src="images/synthesis1.png" width = 80% alt="Synthesis of research results"> --- <h1> Publishable SLR: <span class="subtitle"> To stand out of the review (S4) </span></h1><hr> ⛯ The objective of SLR: To answer the research question.<br> ⛯ Start SLR with research question: Be as specific as possible. <br>   Example 1 = https://doi.org/10.3389/fpsyg.2022.1075818 <br>   Example 2 = https://doi.org/10.1080/17501229.2024.2355296 <br> ⛯ Better format: <img src="images/SLR format.png" width = 75% alt="My SLR format">   Example 1 = https://doi.org/10.1016/j.ijer.2024.102380 <br>   Example 2 = https://doi.org/10.1080/17501229.2024.2355296 --- <h1> Synthesis of the research results </h1><hr> <h8> An example from my work. </h8> <img src="images/synthesis2.png" width = 70% alt="Synthesis of research results" border="1px #555555"> --- <h1> Emphasis from my current work </h1><hr> <h8> <strong>Real objective:</strong> Defining innovation in digital transformation (DT) in higher education</h8><br> <strong>Research Question:</strong> What are the criteria of innovation in DT in (Southeast Asian) higher education?<br> <strong>Framework:</strong> Sternberg's (2003) framework: 5 types of innovation<br><br> <img src="images/my_current_work.png" width = 68% alt="Synthesis of research results" border="1px #555555"> -- <footer style="position:absolute; right: 120px"> Any question so far?</footer> --- <h4> Systematic Literature Review </h4> <h5> Part 2: Writing Process </h5> <h6> Faisal Mustafa </h6> <h6> 2024-05-31 </h6> --- <h1> Introduction section</h1> <hr> <h8> Structure of the Introduction section:</h8> <ol> <li> The significance of the topic.</li> <li> The problem(s) or previous research which makes the topic even more significance/interesting. </li> <li> Previous review (almost) related to the topic, and research gap as CS.</> <li> Why the current study is significant/urgent<sup>*</sup>, and RQ. </li> </ol> <h8> Many articles do not include literature review section as in non SLR papers</p> <footer> CS = closing statement; * it's time to link to the real objective</footer> --- <h1> Method section </h1> <hr> <h8> The method section covers:</h8> <ol> <li> Databases, coverage, and literature search - summarized in PRISMA diagram (<a href="https://www.prisma-statement.org"> https://www.prisma-statement.org</a>) </li> <li> Literature selection procedure<sup>*</sup> and description of final list, summaries by:</li> <ul> <li> Year</li> <li> Research method</li> <li> Country</li> <li> Database </li> <li> Quality</li> <li> etc.</li> </ul> <li> Coding<sup>♣</sup> and analysis procedure, including the framework and how it is used (to achieve real objective).</li> </ol> <h8> Include definition and justification for the selection of database, coverage, keywords (in lit. search), analysis, framework, etc. </h8> <footer> * including quality assessment, ♣ e.g. Thematic Analysis (TA) or Reflexive TA </footer> --- <h1>Results (and discussion) section</h1><hr> <h8> Suggestions for structuring this section:</h8> <ul> <li> Organize the results based on the framework </li> <li> Visualize the data for each framework component </li> <li> If each component consists of many themes, create heading for each component </li> <li> Summarize data and discuss it (what it means and what others say about it)</li> </ul> <h8> <em>Note.</em> This section answers your research question. </h8><br> <img src="images/result.png" width = 30% alt="Structuring research results" border="1px #555555"> <div class="topbottom"> <table class="blueTable" style ="width:670px"> <thead> <tr> <th style="width:15%; font-weight: bold;">Component</th> <th style="width:12%; font-weight: bold;">Themes</th> <th style="width:52%; font-weight: bold;">Studies</th> <th style="width:21%; font-weight: bold;">Remarks <em>(optional)</em></th> </tr> </thead> <tbody> <tr> <td> Component 1 </td><td>Theme 1</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td></td><td>Theme 2</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td></td><td>Theme 3</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td></td><td>Theme 4</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> Component 2 </td><td>Theme 1</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> </td><td>Theme 2</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> </td><td>Theme 3</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> </td><td>Theme 4</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> Component 3 </td><td>Theme 4</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> </td><td>Theme 2</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> <tr> <td> </td><td>Theme 3</td><td> Author1 (2020); Author2 (2013); Author3 (2021), ...</td><td> ... </></tr> </tbody> </table> </div> --- <h1> Conclusion section<sup>*</sup> </h1><hr> <h8> Primary components </h8> <ol> <li> Summarize the results </li> <li> Describe how the results lead to <em> real objective</em></li> <li> Present and discuss <em>the real objective </em> with interesting diagram </em> </ol> <h8> Additional components:</h8> <ol> <li> Further empirical research suggestions based on reseach gap in the results </li> <li> Limitation of the study and future SLR </li> </ol> <footer> * Sometimes "Discussion and Conclusion" or "Discussion" with the additional "Conclusion" section </footer>